Optimization of multiple services via machine learning

ABSTRACT

A method, computer-readable medium, and apparatus for modeling data of a service for providing a policy are disclosed. For example, a method may include a processor for generating a first policy for a first service by a first policy model using machine learning for processing first data of the first service, determining whether the first policy is to be applied to a second service, applying the first policy to the second service when the first policy is deemed to be applicable to the second service, wherein the applying the first policy provides the first policy to a second policy model using machine learning for processing second data of the second service, generating a second policy for the second service, and implementing the second policy in the second service, wherein the first service and the second service are provided by a single service provider.

This application is a continuation of U.S. patent application Ser. No.15/222,653, filed on Jul. 28, 2016, now U.S. Pat. No. 10,719,777, whichis herein incorporated by reference in its entirety.

The present disclosure relates generally to methods, computer-readablemedia and apparatuses for optimizing a service, e.g., generating andimplementing a new policy for the service.

BACKGROUND

It is challenging to ensure that customers are satisfied with a givenservice on an on-going basis due to ever changing conditions. Forexample, a network service provider may provide a cellular service, butchanging conditions may occur, e.g., a network component may fail, a newbandwidth requirement may impact the overall quality of service, a newpopular software application may require additional bandwidth from theunderlying cellular network as large number of subscribers begin usingthe new software application, and the like.

SUMMARY

In one example, the present disclosure discloses a method,computer-readable medium, and apparatus for modeling data of a servicefor providing a policy are disclosed. For example, a method may includea processor for generating a first policy for a first service by a firstpolicy model using machine learning for processing first data of thefirst service, determining whether the first policy is to be applied toa second service, applying the first policy to the second service whenthe first policy is deemed to be applicable to the second service,wherein the applying the first policy provides the first policy to asecond policy model using machine learning for processing second data ofthe second service, generating a second policy for the second service,and implementing the second policy in the second service, wherein thefirst service and the second service are provided by a single serviceprovider.

BRIEF DESCRIPTION OF THE DRAWINGS

The teachings of the present disclosure can be readily understood byconsidering the following detailed description in conjunction with theaccompanying drawings, in which:

FIG. 1 illustrates an example system related to the present disclosure;

FIG. 2 illustrates a flowchart of an example method for modeling data ofa service for providing a policy, according to the present disclosure;

FIG. 3 illustrates a flowchart of an example method for implementing afunction in accordance with a policy generated by a policy model in anautonomous system; and

FIG. 4 illustrates a high-level block diagram of a computing devicespecially configured to perform the functions, methods, operations andalgorithms described herein.

To facilitate understanding, identical reference numerals have beenused, where possible, to designate identical elements that are common tothe figures.

DETAILED DESCRIPTION

The present disclosure broadly discloses methods, computer-readablemedia and apparatuses for optimizing a service, e.g., generating andimplementing a new policy for the service. More specifically, in oneembodiment, the new policy is deduced from a first service and thenpresented to be used by a second service. For example, the new policy isdetermined automatically through the use of machine learning.

As discussed above, it is challenging to ensure that customers aresatisfied with a given service on an on-going basis due to ever changingconditions. For example, a network service provider may provide acellular service, but is not aware that a changing condition hasoccurred, e.g., a new popular software application has gone viral, i.e.,a very large number of subscribers are all using the new softwareapplication at the same time. Unbeknown to the network service provider,there is a sudden surge in demand for additional bandwidth due to thisnew popular software application. The network service provider maydeduce this need eventually through various channels, e.g., customerscontacting a customer care center of the network service provider toinquire why the data service is so congested when using the new softwareapplication, customers leaving comments on a website of the networkservice provider with similar concerns or complaints, and the like.

In one embodiment, machine learning can be deployed to continuallymonitor various data associated with the service such that anomalouspatterns can be quickly determined and analyzed. This use of machinelearning can be useful to detect and address a new problem or a new needfor a resource that the network service provider should address asquickly as possible to ensure customer satisfaction.

Machine learning allows a trained model to learn from and makepredictions on data. Machine learning operates by building the modelfrom an example training set of input observations in order to makepredictions or decisions expressed as outputs, rather than followingstatic programming instructions. In one embodiment, the presentdisclosure employs machine learning in predictive analytics, e.g., todiscover “new insights” through learning from historical relationshipsand trends in the data. Machine learning tasks may encompass differentcategories of learning, e.g., supervised learning (having access tolabelled data), unsupervised learning (having no access to labelleddata), semi-supervised learning (having limited access to labelleddata), and reinforcement learning. It should be noted that any number ofdifferent machine learning approaches can be implemented, e.g., supportvector machines, clustering analysis, manifold learning, parametricdistribution learning, and the like. In one embodiment, reinforcementlearning is subsequently used where reinforcement learning focuses onhow an agent ought to take actions in an environment so as to maximize anotion of a long-term reward, e.g., customer satisfaction, betterperformance in terms of speed and efficiency, faster reaction time,lower cost, higher reliability, better security, and the like.Reinforcement learning methods attempt to find a policy that maps“states” (e.g., services in the present disclosure) to the actions thatthe agent ought to take in those states. Reinforcement learning differsfrom the supervised learning in that correct input/output pairs are notpresented, nor sub-optimal actions explicitly corrected.

However, another aspect to consider is that a service provider, e.g., anetwork service provider, may engage in providing a plurality ofservices instead of a single service. For example, a network serviceprovider may provide a landline telephony service (e.g., local phoneservice, long distance phone service, facsimile service, or Voice overInternet Protocol (IP) service), a cellular service (e.g., cellularphone service), a data service (e.g., broadband access service), amultimedia delivery service (e.g., television, movie or any otherprogramming delivery services), a connected car service (e.g., providingdata or cellular service to a customer's vehicle), a connected premisesservice (e.g., providing data or cellular service to a customer's homeor business for managing various aspects of the premises, such asheating, cooling, lighting, security monitoring and the like), and soon. Each of these provided services is quite different from service toservice. As such, machine learning can be deployed for each of theservices to detect and address its respective new problem or new needfor a resource that the network service provider should address asquickly as possible to ensure customer satisfaction.

In one embodiment of the present disclosure, the knowledge acquiredthrough machine learning in one service is used to optimize theperformance of another different service. For example, machine learningsuch as reinforcement learning (RL) is used for resource optimizationfor a plurality of services of a service provider such that thelearnings from individual services can be used to optimally impact theother services. In addition to the increase in overall quality acrossthe multiple services, one additional advantage is the strengthening ofthe service provider's overall brand. For example, the serviceprovider's overall brand can also be improved due to its consistentresponse in addressing a problem or need, its consistent timely response(e.g., within one day, within two days and so on) in addressing aproblem or need, its consistent response format or modality (e.g., howthe response is communicated, e.g., email, text messaging, phone calls,social network postings, and so on) in addressing a problem or need, itsconsistent method of response (e.g., deploying new resources, providinga credit to customers, providing a free service to customers, sending atechnician to troubleshoot the problem and so on) in addressing aproblem or need, and so on.

One aspect the present disclosure is that the present embodiments arescalable to address large data analytics with minimal human interventionin terms of identifying one or more new policies, e.g., performanceindicators and possible strategies to effect a positive change, tofacilitate better user experience, and at the same time reducing costincurred. Furthermore, adaptive mechanisms can be used that woulddynamically learn over time for transferring existing knowledge of oneor more services to other services, e.g., existing services or even newservices (i.e., services that have yet to be deployed). This couldresult in a mutually beneficial environment where customers mayexperience better performance, before even noticing performance issuesif any, since the method may use prior knowledge to interpret andpredict potential issues/events.

In one embodiment of the present disclosure, the present method providesa cohesive integration of relevant services of a service provider byutilizing machine learning, e.g., Reinforcement Learning (RL), bydefining “states” as the different services of a service provider, e.g.,such as telephony service, cellular or mobility service, data service,multimedia delivery service, connected car service, connected premisesservice, and so on. In addition, “actions” can be defined as the driversof the relevant components pertaining to these states such as actionstaken for addressing cost, actions taken for addressing reliability,actions taken for addressing time of delivery, actions taken foraddressing customer inquiries or complaints, actions taken foraddressing network performance, actions taken for addressing quality ofservice, and so on. Finally, “policies” are defined that drive the“state-action” space holistically such that the information gatheredfrom all these various services are optimally utilized to define, inturn, a new set of one or more policies that would further maximize areward to each of the services. The term “reward” is broadly defined toencompass a broad range of possible goals for a service, e.g.,increasing performance of a service, reducing operating cost for aservice, reducing cost to customers of a service, increasing reliabilityof a service, increasing security of a service, increasing customersatisfaction for a service, reducing the time to address customer'sconcerns or complaints, and so on.

To illustrate, different services/states of a service provider willshare some set of drivers/actions. Thus, a policy model using machinelearning will learn from one state that could impact other states aswell. In such a schema, RL may effectively leverage prior informationobtained from these state-action spaces to chart out a set of policiesthat would maximize the reward for the services individually as well asholistically. To further illustrate, an example is where datameasurements such as chat emotions, call length and user interactionhistory can be leveraged usefully for multiple tasks such as customersatisfaction estimation, predicting churn and willingness to recommend.In other words, the results of customer satisfaction estimation, rootcause analysis, predicting churn and willingness to recommend are allimportant aspects of any services, irrespective as to what services arebeing provided. This example illustrates that these data measurementscould have effects across services/products of a single serviceprovider, which when analyzed properly could have a significant impacton the overall brand value of the service provider.

In addition to lowering costs, which would otherwise have been incurredfor each service unit performing its own modeling, the present methodwill help improve customer experience by addressing issues which mayhave yet to reach a tipping point for the users to file a complaint. Inother words, by using the present modeling with machine learning, apotential outlier pattern in one particular service can be preemptivelydetected from other multiple services prior to the outlier patternreaching a critical point in the particular service where customersatisfaction will be impacted.

As discussed below, although the present disclosure may initially startwith existing data (e.g., used for training) to model existing servicesand actions in isolation, a machine learning framework such as an RLframework would then subsequently use them at once to output a set ofone or more policies that would optimally use the available informationto maximize the final reward for a plurality of services. Severaladvantages are provided by the present disclosure, e.g., the ability toadapt with changing data and the potential addition of new services (andpossibly deletion of some existing services), as information containedin the learned policies can be reconfigured. It is also expected thatthe present disclosure will improve customer experience (e.g., bysolving problems before the problems become noticeable) and will reducecost (e.g., by a holistic treatment across different services). Theseand other aspects of the present disclosure are described in greaterdetail below in connection with the discussion of FIGS. 1-4.

To better understand the present disclosure, FIG. 1 illustrates anexample network, or system 100 suitable for implementing embodiments ofthe present disclosure for modeling data of a service for providing apolicy. In one example, the system 100 comprises a Long Term Evolution(LTE) network 101, an IP network 113, and a core network 115, e.g., anIP Multimedia Subsystem (IMS) core network. In one example, system 100is provided and operated by a single network operator or network serviceprovider. FIG. 1 also illustrates various user endpoint devices, e.g.,LTE user endpoint devices 116 and 117. The user mobile endpoint devices116 and 117 may each comprise a cellular telephone, a smartphone, atablet computing device, a laptop computer, a pair of computing glasses,a wireless enabled wristwatch, or any other cellular-capable mobiletelephony, a device deployed in a vehicle, and computing device(broadly, “mobile endpoint devices”). In another embodiment, the userendpoint devices 116 and 117 may be stationary devices, e.g., set topboxes, home gateways, security panels at a premises, home appliances,Internet of Things (loT) sensors, and the like. For the purposes ofproviding illustrated examples, endpoint devices 116 and 117 will bedeemed to be mobile devices in various examples discussed below, but itshould be noted that endpoint devices 116 and 117 can be both mobiledevices and/or stationary devices.

In one embodiment, each of the user mobile endpoint devices is capableof executing one or more mobile software applications, e.g., softwareapplications for transmitting and/or receiving multimedia content,gaming, shopping, surfing the web, sending and receiving data, sendingand receiving messages such as emails and text messages, implementingcall sessions such as voice over IP calls, video conferencing, and thelike.

In one example, the LTE network 101 comprises an access network 103 anda core network 105. In one example, the access network 103 comprises anevolved Universal Terrestrial Radio Access Network (eUTRAN). The eUTRANsare the air interfaces of the 3rd Generation Partnership Project (3GPP)LTE specifications for mobile networks. In one example, the core network105 comprises an Evolved Packet Core (EPC) network. An EPC networkprovides various functions that support wireless services in the LTEenvironment. In one example, an EPC network is an Internet Protocol (IP)packet core network that supports both real-time and non-real-timeservice delivery across a LTE network, e.g., as specified by the 3GPPstandards. In one example, all eNodeBs in the access network 103 are incommunication with the EPC network 105. In operation, LTE user equipmentor user endpoints (UE) 116 may access wireless services via the eNodeB111 and the LTE UE 117 may access wireless services via the eNodeB 112located in the access network 103. It should be noted that any number ofeNodeBs can be deployed in an eUTRAN. In one illustrative example, theaccess network 103 may comprise one or more eNodeBs.

In EPC network 105, network devices Mobility Management Entity (MME) 107and Serving Gateway (SGW) 108 support various functions as part of theLTE network 101. For example, MME 107 is the control node for the LTEaccess-network. In one embodiment, it is responsible for UE (UserEquipment) tracking and paging (e.g., such as retransmissions), beareractivation and deactivation process, selection of the SGW, andauthentication of a user. In one embodiment, SGW 108 routes and forwardsuser data packets, while also acting as the mobility anchor for the userplane during inter-eNodeB handovers and as the anchor for mobilitybetween LTE and other wireless technologies, such as 2G and 3G wirelessnetworks.

In addition, EPC (common backbone) network 105 may comprise a HomeSubscriber Server (HSS) 109 that contains subscription-relatedinformation (e.g., subscriber profiles), performs authentication andauthorization of a wireless service user, and provides information aboutthe subscriber's location. The EPC network 105 may also comprise apublic data network (PDN) gateway 110 which serves as a gateway thatprovides access between the EPC network 105 and various data networks,e.g., other IP networks 113, an IMS core network 115, and the like. Thepublic data network gateway is also referred to as a PDN gateway, a PDNGW or a PGW.

The EPC network 105 may also include an application server (AS) 190. Inone embodiment, AS 190 may comprise a computing system, such ascomputing system 400 depicted in FIG. 4, and may be configured toprovide one or more functions (e.g., implementing a policy model) formodeling mobile traffic for providing a policy, and for performingvarious other operations in accordance with the present disclosure.Accordingly, the AS 190 may be connected directly or indirectly to anyone or more network elements of EPC network 105, and of the system 100in general, that are configured to gather and forward network analyticinformation, such as signaling and traffic data, and other informationand statistics to AS 190 and to receive instructions from AS 190. Inaddition, AS 190 may be configured to receive requests to implement oneor more actions or functions as discussed below after modeling mobiletraffics using machine learning. For example, a policy model employingmachine learning (broadly an analytic engine) can be implemented in AS190 for processing mobile traffic. AS 190 may be further configured toperform other functions such as those described below in connection withthe example methods 200 and 300 of FIGS. 2 and 3.

In one example, AS 190 may be deployed in a network operations center(NOC) of a cellular network operator, e.g., an entity operating the EPCnetwork 105, LTE network 101, access network 103, and so on. Due to therelatively large number of connections available between AS 190 andother network elements, none of the actual links to the applicationserver are shown in FIG. 1. Similarly, links between MME 107, SGW 108,broadcast server 194, eNodeBs 111 and 112, PDN gateway 110, and othercomponents of system 100 are also omitted for clarity.

It should be noted that the LTE network 101 is disclosed to provide avery brief summary description of the underlying framework that isutilized to provide a cellular or mobility service. Similarly, variousother networks 170 having respective application servers 175 can also bedeployed by a single service provider, e.g., a network service providerin providing a plurality of other services, e.g., telephony services,data services, multimedia delivery services, connected car services,connected premises services, and so on. For clarity reasons, theunderlying framework for these other networks 170 are not shown in FIG.1, but it is understood that a single network service provider iscapable of providing two or more of these services.

As such, the foregoing description of the system 100 is provided as anillustrative example only. In other words, the example of system 100 ismerely illustrative of one network configuration that is suitable forimplementing embodiments of the present disclosure. As such, otherlogical and/or physical arrangements for the system 100 may beimplemented in accordance with the present disclosure. For example, AS190, broadcast server 194 and/or other network components may bedeployed in core network 115 instead of being deployed within the EPCnetwork 105, or in other portions of system 100 that are not shown,while providing essentially the same functionality. For example, thefunctionality of AS 190 for a cellular service can be implemented viathe application server 120 having an analytical engine 121 utilizingdatabase 122 to store various data associated with the mobile trafficfor the cellular service. In fact, in one embodiment the applicationserver 120 is configured as a dedicated policy modeling server forimplementing one or more policy models using machine learning to supporta plurality of different services offered by the same network serviceprovider. For example, the policy models for a plurality of differentservices, e.g., telephony services, cellular services, data services,multimedia delivery services, connected car services, and connectedpremises services, can all be instantiated in the AS 120 (which mayencompass a plurality of application servers to handle the large volumeservice data).

In addition, although aspects of the present disclosure have beendiscussed above in the context of a long term evolution (LTE)-basedwireless network, examples of the present disclosure are not so limited.Thus, the teachings of the present disclosure can be applied to othertypes of wireless networks (e.g., 2G network, 3G network and the like),for modeling mobile traffic for providing a policy. In fact, the abovelisting of various services should not be deemed to be an exhaustivelisting of services. Thus, these and other modifications are allcontemplated within the scope of the present disclosure.

The present disclosure uses a method that models a plurality of services(e.g., two of more services) offered by a single service provider, e.g.,a single network service provider for providing network based servicesas discussed above, for identifying one or more policies that should beimplemented for across multiple services. In doing so, the presentdisclosure offers an autonomous method that is capable of leveraginglearned patterns in one service to be applied to a different service.

FIG. 2 illustrates a flowchart of an example method 200 for modelingdata of a service for providing a policy. In one embodiment, the steps,operations or functions of the method 200 may be performed by any one ormore of the components of the system 100 depicted in FIG. 1. Forexample, in one embodiment, the method 200 is performed by theapplication server (AS) 190, 175 or 120. In another embodiment, themethod 200 is performed by AS 190 in coordination with other componentsof the system 100, such as broadcast server 194 (for broadcastingvarious notifications), eNodeBs 111 and 112, and so forth.Alternatively, or in addition, one or more steps, operations orfunctions of the method 200 may be implemented by a computing devicehaving a processor, a memory and input/output devices as illustratedbelow in FIG. 4, specifically programmed to perform the steps, functionsand/or operations of the method. Although any one of the elements insystem 100 of FIG. 1 may be configured to perform various steps,operations or functions of the method 200, the method will now bedescribed in terms of an embodiment where steps of the method areperformed by a processor, such as processor 402 in FIG. 4. For example,the processor may comprise a processor of a dedicated application serverof a single network operator configured to model data of a service forproviding a policy.

The method 200 begins in step 205 and proceeds to step 210. In step 210,the processor creates a model, e.g., a policy model, for modeling dataof a service for providing a policy. Step 210 can be deemed as a partialpre-processing step that is performed offline. However, once the modelis created and trained, the model can be dynamically retrained andupdated. For example, the model can be trained on data associated with aparticular type of service, such as a cellular service for generating apolicy (e.g., a learned policy) predicted from various “state-action”relationships. For example, the “state” can be represented by theservice itself or alternatively can be represented by a plurality of“sub-states” relating to various broad categories associated with theservice, e.g., “billing associated with the service,” “cost associatedwith the service,” “performance associated with the service,”“reliability associated with the service,” “security associated with theservice,” “timely delivery of product or services associated with theservice,” “customer care associated with the service,” and the like. Inaddition, “actions” can be defined as the drivers of the relevantcomponents pertaining to these states or sub-states such as actionstaken for addressing billing (e.g., adjusting the billing cycle for acustomer, removing a charge from a bill questioned by a customer, andthe like), actions taken for addressing cost (e.g., reducing the cost ofa service charged to a customer, reducing a cost in providing theservice, offering a new plan to reduce cost for the customer and thelike), actions taken for addressing network performance (e.g., sending atest signal to measure performance of a network based component,increasing or instantiating new network based resources, decreasing orremoving new network based resources, and the like), actions taken foraddressing reliability (e.g., deploying redundant network resources,monitoring maintenance schedules, ensuring replacement of poorlyperforming network components, and the like), actions taken foraddressing security (e.g., maintaining firewalls, updating firewallrules and filters, monitoring for malicious behaviors and attacks, andthe like), actions taken for addressing time of delivery (e.g.,monitoring scheduled delivery of products and services, monitoringtimely onsite visits by technicians, and the like), actions taken foraddressing customer inquiries or complaints (e.g., answering concerns orcomplaints of customers, following up with customers to ensure concernsand complaints are addressed to the customers' satisfaction, determiningwhether the solutions offered to the customers are appropriate and thelike), actions taken for addressing quality of service (e.g., monitoringwhether contracted QoS levels are met for each customer, and the like)and so on.

In one embodiment, the policy model can be provided with a training setof data for a particular type of service. For example, the dataassociated with a cellular service can be obtained from customer carerecords, maintenance records, operations records such as call detailrecords, marketing records, customers surveys, and the like. Such datacan be categorized into “state-action” relationships that will beapplied as training data to the policy model that employs machinelearning. In one embodiment, the policy model will provide one or morepolicies (e.g., a long term reward) that are learned from the trainingdata. For example, the policies may relate to reducing cost (e.g., forcustomers and/or the service provider), improving the customer'sexperience, reconfiguring of network based or customer based equipmentdue to changing conditions, suggesting a new feature for a service,suggesting an entirely new service, presenting an alternate solution toan existing problem, detecting a potentially new problem that has yet tobe detected (e.g., a breach in the network security, a pending networkfailure, and the like). In sum, the policy model is able to learninformation across the state-action space to improve various drivers forthe service.

In one embodiment, the training data can also be tagged to identify oneor more service features, such as privacy feature, security feature,billing feature, performance feature, safety feature and the like. Forexample, each customer care record may contain a field that identifiesthe concern of the caller, e.g., the caller is calling to complain abouta privacy issue, a security issue, a billing issue, a performance issue,a safety issue and the like. Such tagging of the training data willallow the policy model via machine learning to be able to categorize andextract one or more service features. In other words, the policy modelis able to associate each “state-action” relationship with one or moreservice features.

In one embodiment, once the policy model is trained in step 210, newfirst data in step 215 can be continuously provided to the policy modelfor analysis and/or subsequent retraining. In other words, the policymodel is not static and instead is able to be continuously anddynamically retrained with new data. This approach allows the policymodel to have the capability to learn and predict new patterns that mayforecast potentially new problems that the policy model has yet todetect and encounter.

In step 220, the processor generates a “first” policy for the “first”service. It should be noted that the terms “first” and “second” are usedherein to assist the reader to distinguish between different policiesand different services and are not intended to impart any limitations tothe present disclosure. In fact, there may be additional policies andservices greater than the quantity of two (2) as illustrativelydiscussed in FIG. 2. For example, the machine learning of the policymodel may deduce from the state-action space that certain actions takenby the service provider resulted in meeting or maximizing a reward. Toillustrate, the machine learning may deduce from a large number ofcustomer care records that customers who received a follow up telephonecall from a live customer care agent were very unlikely to drop theservice when compared to customers who did not receive a follow uptelephone call from the live customer care agent for the same problem.The policy model may then generate a new policy for a customer carecenter to require all customer care agents to call any customer who hasexperienced this type of problem.

In step 230, the processor implements the first policy. In one example,the first policy is automatically implemented. For example, if the firstpolicy relates to a customer care issue, the first policy can beimplemented as a new guideline to be followed by all customer careagents. Alternatively, the first policy can be implemented via anautonomous system, e.g., an interactive voice response (IVR) system. Forexample, an IVR system can be automatically configured to dial back acustomer who had previously experienced a particular problem inquiringwhether the customer wishes to speak to a live agent if the problem hasnot been resolved to his or her satisfaction. When the customerindicates that a subsequent discussion to a live agent is desired, thenthe IVR system will connect the customer to a live customer care agent.In this example, the generated policy is translated into a feature orfunction of an autonomous system. Such approach will greatly increasethe ability to rapidly update features and functions of autonomoussystems to address trending issues that are detected by the policymodel. It should be noted that the autonomous system is not limited toan IVR system for a customer care system. For example, the autonomoussystem may encompass a trouble ticketing system, a network provisioningsystem, a network maintenance scheduling system, a network resourceinstantiation controller for a software defined network, and the like.

In step 240, the processor determines whether the first policy isapplicable to a second service. For example, the first service may be acellular service and the second service may be a multimedia contentdelivery service. In one example, the first policy may entail suggestingthat customers who experienced a particular problem should receive afollow up telephone call from a live customer care agent. In thisexample, the problem may be an interrupted service, e.g., a droppedcellular call that occurred in response to a particular scenario. Theprocessor will determine whether this policy is relevant to anotherdifferent service. Since the multimedia content delivery service mayalso experience service interruption, then the first policy will likelyproduce the same reward projected for the first service as in the secondservice. It should be noted that it is not required that the firstpolicy must have a direct correlation with the second service before thefirst policy is applied to the second service. For example, the firstpolicy may relate to sending a text message to a customer's cell phone.However, if the second service relates to a service monitoring HVACsystems via thermostat panels at the customers' sites, then the policyof sending a text message to a customer's thermostat panel would atfirst glance to be inapplicable since a thermostat panel is not capableof sending or receiving a text message. In contrast to such approach,this first policy may actually cause the second policy model for theservice monitoring HVAC systems to possibly put forth a proposed newfeature to be offered to its customers, e.g., providing text messagingcapability to the thermostat panels. In one embodiment, the decisionwhether to send the first policy to a second service can be based on thetagged service features associated with the first policy. For example,if the first policy relates to a security or a safety service feature,then the first policy will always be sent to the second service and soon. Such decision can be tailored by the service provider as to howservice features will be weighed to provide the decision. Returning tostep 240, if the determination is positive, method 200 proceeds to step250, otherwise the method 200 returns to step 210.

In step 250, the processor applies the first policy to the trainedsecond policy model of the second service. Similar to step 210, thesecond service will also have a trained policy model similar to thefirst service while receiving new second data for the second service instep 255. However, unlike step 210, the processor in step 250 will alsoapply the first policy in the second policy model of the second service.This will allow the machine learning of the second service to leveragethe new first policy deduced for the first service. This allows multipleservices offered by a single service provider to cross pollinateknowledge learned from one service to be quickly applied to a seconddifferent service.

In step 260, the processor generates a “second” policy for the secondservice. For example, the machine learning of the second policy modelmay deduce from its own state-action space and the first policy thatcertain actions taken by the service provider may result in meeting ormaximizing a reward. To illustrate, the machine learning may deduce fromthe first policy and a small number of customer care records thatcustomers who received a follow up telephone call from a live customercare agent were very unlikely to drop the service when compared tocustomers who did not receive a follow up telephone call from the livecustomer care agent for the same problem. The second policy model maythen generate a new second policy for a customer care center to requireall customer care agents to call any customer who has experienced thistype of problem. For example, the customer care records for the secondservice may only have a limited experience on this issue, but combinedwith the first policy, it may quickly arrive at a similar policy to beimplemented in the second service. As this example illustrates, a policymodel of a second service using machine learning can be configured tolearn from another state to rapidly enhance its own performance.

In step 270, the processor implements the second policy. In one example,the second policy is automatically implemented. For example, if thesecond policy relates to a customer care issue, the second policy can beimplemented as a new guideline to be followed by all customer careagents. Alternatively, the second policy can be implemented via anautonomous system, e.g., an interactive voice response (IVR) system. Forexample, an IVR system can be automatically configured to dial back acustomer who had previously experienced a particular problem inquiringwhether the customer wishes to speak to a live agent if the problem hasnot been resolved to his or her satisfaction. When the customerindicates that a subsequent discussion to a live agent is desired, thenthe IVR system will connect the customer to a live customer care agent.In this example, the generated second policy is translated into afeature or function of an autonomous system. Such approach will greatlyincrease the ability to rapidly update features and functions ofautonomous systems to address trending issues that are detected by thepolicy model. Again, the autonomous system may encompass a troubleticketing system, a network provisioning system, a network maintenancescheduling system, a network resource instantiation controller for asoftware defined network, and the like.

In step 280, the processor determines whether the second policy isapplicable to another service, e.g., the first service or a thirdservice (not shown). It should be noted that the first policy may be thesame or different from the second policy depending on the machinelearning of the policy model of the respective first and secondservices. If the first policy is the same as the second policy, then themethod will ends in step 295, otherwise the method will return to step210, where the second policy is provided to the first policy model. Forexample, the second policy may suggest sending a visual text to bepresented on a television display of the customer since the secondservice provides multimedia content to its customers. Thus, calling thecustomers may not be the best communication modality for the secondservice. In other words, the second service applied the knowledge of thefirst policy through its policy model, but instead deduce for itself aslightly different policy that is more appropriate for the secondservice. In turn, this second policy can be provided back to the firstservice which may or may not be impacted by the knowledge of this secondpolicy.

FIG. 3 illustrates a flowchart of an example method 300 for implementinga function in accordance with a policy generated by a policy model in anautonomous system. In one embodiment, the steps, operations or functionsof the method 300 may be performed by any one or more of the componentsof the system 100 depicted in FIG. 1. For example, in one embodiment,the method 300 is performed by one of mobile endpoint devices 116 or117. In another embodiment, the method 300 is performed by othercomponents of the system 100, such as application server 190, 175 or120, and so forth. Alternatively, or in addition, one or more steps,operations or functions of the method 300 may be implemented by acomputing device having a processor, a memory and input/output devicesas illustrated below in FIG. 4, specifically programmed to perform thesteps, functions and/or operations of the method. Although variouselements in system 100 of FIG. 1 may be configured to perform varioussteps, operations or functions of the method 300, the method will now bedescribed in terms of an embodiment where steps of the method areperformed by a processor, such as processor 402 in FIG. 4. For example,the processor may comprise a processor of a mobile endpoint device or anapplication server.

The method 300 begins in step 305 and proceeds to step 310. In step 310,the processor outputs a policy for a service. For example, step 310 issimilar to step 220 or step 250 of FIG. 2.

In step 320, the processor generates or modifies a function inaccordance with the policy. For example, if the policy is to provide anew function in a customer care system, e.g., an IVR system, then method300 may utilize similar software codes to provide a new functionconsistent with the policy. For example, if an IVR system already has areminder function for customers who are late with their scheduledpayments (e.g., calling each customer with a prerecorded notification topay a bill), such reminder function can be modified slightly to providea new “follow-up” function (e.g., calling each customer with aprerecorded notification to inquire whether a previously experiencedproblem has been resolved to his or her satisfaction). The only changecan simply be the prerecorded message and the triggering event forsending the notification. In another example, if the policy indicatesthat a previously scheduled maintenance or update of a network componentneeds to be accelerated given a current level of complaints, the methodmay alter the maintenance schedule directly and inform a maintenancesupervisor of the change. In another example, if the policy indicatesthat a firewall filter needs an immediate update, the method may alterthe firewall filter directly and inform a network security supervisor ofthe change. In this manner, any number of autonomous systems can beautomatically updated and reconfigured with a newly generated ormodified function based on the generated policy.

In step 330, the processor implements the function in the autonomoussystem. For example, the method deploys the newly generated or modifiedfunction in the autonomous system. Method 300 then ends in step 395.

Thus, the present disclosure provides advances in the field of networkmanagement and/or autonomous system modification using machine learning.By converting deduced policies intended for one service to be used byanother different service, a policy model using machine learning willlearn valuable knowledge from one state that could impact other states.The advantage of the present disclosure allows a single service providerto leverage knowledge learned from one service to be applied rapidly toanother service. Additionally, such deduced policy can be used directlyto update or reconfigure function of an autonomous system to bring abouta rapid response to changing conditions that may impact the variousservices provided by a single service provider.

FIG. 4 depicts a high-level block diagram of a computing device suitablefor use in performing the functions described herein. As depicted inFIG. 4, the system 400 comprises one or more hardware processor elements402 (e.g., a central processing unit (CPU), a microprocessor, or amulti-core processor), a memory 404 (e.g., random access memory (RAM)and/or read only memory (ROM)), a module 405 for modeling data of aservice for providing a policy, and various input/output devices 406(e.g., storage devices, including but not limited to, a tape drive, afloppy drive, a hard disk drive or a compact disk drive, a receiver, atransmitter, a speaker, a display, a speech synthesizer, an output port,an input port and a user input device (such as a keyboard, a keypad, amouse, a microphone and the like)). Although only one processor elementis shown, it should be noted that the computing device may employ aplurality of processor elements. Furthermore, although only onecomputing device is shown in the figure, if the method 200 or the method300 as discussed above is implemented in a distributed or parallelmanner for a particular illustrative example, i.e., the steps of themethod, or the entire method is implemented across multiple or parallelcomputing devices, then the computing device of this figure is intendedto represent each of those multiple computing devices.

Furthermore, one or more hardware processors can be utilized insupporting a virtualized or shared computing environment. Thevirtualized computing environment may support one or more virtualmachines representing computers, servers, or other computing devices. Insuch virtualized virtual machines, hardware components such as hardwareprocessors and computer-readable storage devices may be virtualized orlogically represented. The one or more hardware processors 402 can alsobe configured or programmed to cause other devices to perform one ormore operations as discussed above. In other words, the one or morehardware processors 402 may serve the function of a controller directingother devices to perform the one or more operations as discussed above.

It should be noted that the present disclosure can be implemented insoftware and/or in a combination of software and hardware, e.g., usingapplication specific integrated circuits (ASIC), a programmable gatearray (PGA) including a Field PGA, or a state machine deployed on ahardware device, a computing device or any other hardware equivalents,e.g., computer readable instructions pertaining to the method discussedabove can be used to configure a hardware processor to perform thesteps, functions and/or operations of the above disclosed methods. Inone embodiment, instructions and data for the present module or process405 for modeling data of a service for providing a policy (e.g., asoftware program comprising computer-executable instructions) can beloaded into memory 404 and executed by hardware processor element 402 toimplement the steps, functions or operations as discussed above inconnection with the illustrative methods 200 and/or 300. Furthermore,when a hardware processor executes instructions to perform “operations,”this could include the hardware processor performing the operationsdirectly and/or facilitating, directing, or cooperating with anotherhardware device or component (e.g., a co-processor and the like) toperform the operations.

The processor executing the computer-readable or software instructionsrelating to the above described methods can be perceived as a programmedprocessor or a specialized processor. As such, the present module 405for modeling data of a service for providing a policy (includingassociated data structures) of the present disclosure can be stored on atangible or physical (broadly non-transitory) computer-readable storagedevice or medium, e.g., volatile memory, non-volatile memory, ROMmemory, RAM memory, magnetic or optical drive, device or diskette andthe like. Furthermore, a “tangible” computer-readable storage device ormedium comprises a physical device, a hardware device, or a device thatis discernible by the touch. More specifically, the computer-readablestorage device may comprise any physical devices that provide theability to store information such as data and/or instructions to beaccessed by a processor or a computing device such as a computer or anapplication server.

While various embodiments have been described above, it should beunderstood that they have been presented by way of example only, and nota limitation. Thus, the breadth and scope of a preferred embodimentshould not be limited by any of the above-described exemplaryembodiments, but should be defined only in accordance with the followingclaims and their equivalents.

What is claimed is:
 1. A device comprising: a processor; and acomputer-readable medium storing instructions which, when executed bythe processor, cause the processor to perform operations, the operationscomprising: generating a first policy for a first service of a networkby a first policy model for the first service using machine learning forprocessing first data of the first service, wherein the first policydefines a first action of the first service; determining whether thefirst policy is to be applied to a second service of the network,wherein each of the first service and the second service comprises atleast one of: a landline telephony service, a cellular service, a dataservice, a multimedia delivery service, a connected car service, or aconnected premises service, wherein the first service and the secondservice are different; applying the first policy to the second servicewhen the first policy is deemed to be applicable to the second service,wherein the applying the first policy provides the first policy to asecond policy model for the second service using machine learning forprocessing second data of the second service; generating a second policyfor the second service, wherein the second policy defines a secondaction of the second service; and implementing the second policy in thesecond service, wherein the implementing the second policy in the secondservice comprises generating a function for an autonomous systemsupporting the second service, wherein the function is inferred from atleast one service feature of the first service via the first policy,wherein the at least one service feature comprises at least one of: aprivacy feature, a security feature, a billing feature, a performancefeature, or a safety feature, and wherein the first service and thesecond service are provided by a single service provider.
 2. The deviceof claim 1, the operations further comprising: determining whether thesecond policy is to be applied to the first service; and applying thesecond policy to the first service when the second policy is deemed tobe applicable to the first service, wherein the applying the secondpolicy provides the second policy to the first policy model.
 3. Thedevice of claim 2, wherein the determining whether the second policy isto be applied to the first service is based on a service feature of thesecond policy.
 4. The device of claim 1, wherein the determining whetherthe first policy is to be applied to the second service is based on atleast one of the at least one service feature of the first policy. 5.The device of claim 1, wherein the autonomous system comprises at leastone of: a trouble ticketing system, a network provisioning system, anetwork maintenance scheduling system, or a network resourceinstantiation controller for the network, wherein the network comprisesa software defined network.
 6. The device of claim 1, wherein themachine learning comprises reinforcement learning.
 7. The device ofclaim 1, wherein the generating the second policy for the second servicecomprises generating the second policy for the second service based onat least the first policy and the second policy model.
 8. A methodcomprising: generating, by a processor, a first policy for a firstservice of a network by a first policy model for the first service usingmachine learning for processing first data of the first service, whereinthe first policy defines a first action of the first service;determining, by the processor, whether the first policy is to be appliedto a second service of the network, wherein each of the first serviceand the second service comprises at least one of: a landline telephonyservice, a cellular service, a data service, a multimedia deliveryservice, a connected car service, or a connected premises service,wherein the first service and the second service are different;applying, by the processor, the first policy to the second service whenthe first policy is deemed to be applicable to the second service,wherein the applying the first policy provides the first policy to asecond policy model for the second service using machine learning forprocessing second data of the second service; generating, by theprocessor, a second policy for the second service, wherein the secondpolicy defines a second action of the second service; and implementing,by the processor, the second policy in the second service, wherein theimplementing the second policy in the second service comprisesgenerating a function for an autonomous system supporting the secondservice, wherein the function is inferred from at least one servicefeature of the first service via the first policy, wherein the at leastone service feature comprises at least one of: a privacy feature, asecurity feature, a billing feature, a performance feature, or a safetyfeature, and wherein the first service and the second service areprovided by a single service provider.
 9. The method of claim 8, furthercomprising: determining, by the processor, whether the second policy isto be applied to the first service; and applying, by the processor, thesecond policy to the first service when the second policy is deemed tobe applicable to the first service, wherein the applying the secondpolicy provides the second policy to the first policy model.
 10. Themethod of claim 9, wherein the determining whether the second policy isto be applied to the first service is based on a service feature of thesecond policy.
 11. The method of claim 8, wherein the determiningwhether the first policy is to be applied to the second service is basedon at least one of the at least one service feature of the first policy.12. The method of claim 8, wherein the autonomous system comprises atleast one of: a trouble ticketing system, a network provisioning system,a network maintenance scheduling system, or a network resourceinstantiation controller for the network, wherein the network comprisesa software defined network.
 13. The method of claim 8, wherein themachine learning comprises reinforcement learning.
 14. The method ofclaim 8, wherein generating the second policy for the second servicecomprises generating the second policy for the second service based onat least the first policy and the second policy model.
 15. A tangiblecomputer-readable medium storing instructions which, when executed by aprocessor, cause the processor to perform operations, the operationscomprising: generating a first policy for a first service of a networkby a first policy model for the first service using machine learning forprocessing first data of the first service, wherein the first policydefines a first action of the first service; determining whether thefirst policy is to be applied to a second service of the network,wherein each of the first service and the second service comprises atleast one of: a landline telephony service, a cellular service, a dataservice, a multimedia delivery service, a connected car service, or aconnected premises service, wherein the first service and the secondservice are different; applying the first policy to the second servicewhen the first policy is deemed to be applicable to the second service,wherein the applying the first policy provides the first policy to asecond policy model for the second service using machine learning forprocessing second data of the second service; generating a second policyfor the second service, wherein the second policy defines a secondaction of the second service; and implementing the second policy in thesecond service, wherein the implementing the second policy in the secondservice comprises generating a function for an autonomous systemsupporting the second service, wherein the function is inferred from atleast one service feature of the first service via the first policy,wherein the at least one service feature comprises at least one of: aprivacy feature, a security feature, a billing feature, a performancefeature, or a safety feature, and wherein the first service and thesecond service are provided by a single service provider.
 16. Thetangible computer-readable medium of claim 15, the operations furthercomprising: determining whether the second policy is to be applied tothe first service; and applying the second policy to the first servicewhen the second policy is deemed to be applicable to the first service,wherein the applying the second policy provides the second policy to thefirst policy model.
 17. The tangible computer-readable medium of claim16, wherein the determining whether the second policy is to be appliedto the first service is based on a service feature of the second policy.18. The tangible computer-readable medium of claim 15, wherein thedetermining whether the first policy is to be applied to the secondservice is based on at least one of the at least one service feature ofthe first policy.
 19. The tangible computer-readable medium of claim 15,wherein the machine learning comprises reinforcement learning.
 20. Thetangible computer-readable medium of claim 15, wherein generating thesecond policy for the second service comprises generating the secondpolicy for the second service based on at least the first policy and thesecond policy model.